Quick start
pip install torch transformers
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizer repo = "LoNebula/data-engineering-mentor"tokenizer = AutoTokenizer.from_pretrained(repo)model = AutoModelForCausalLM.from_pretrained(repo)messages = [ {"role": "system", "content": "You are a practical data engineering mentor. Explain trade-offs and validation steps."}, {"role": "user", "content": "How do I make an incremental model safe to rerun?"},]prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)inputs = tokenizer(prompt, return_tensors="pt").to(next(model.parameters()).device)with torch.inference_mode(): output = model.generate(**inputs, max_new_tokens=220, do_sample=False)print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
Training data
The model uses LoNebula/data-engineering-mentor-dataset, an original 50-example Q&A starter corpus written for this project. It does not copy or redistribute a third-party Q&A dataset. Answers were composed as concise explanations and link to relevant official documentation for further reading. The dataset is published under CC BY 4.0; retain attribution when redistributing the dataset or derived material.
Base model and method
- Base: Qwen/Qwen2.5-0.5B-Instruct
- Method: LoRA supervised fine-tuning, merged into the base weights for direct Transformers loading.
- Dataset: 50 original Q&A pairs, deterministic 85/15 train/test split (42 train, 8 test).
- The model repository follows the base model's Apache-2.0 license. The separately published training dataset is CC BY 4.0; retain its source attribution when reusing or redistributing that dataset.
Intended use and limitations
Use it as a study companion for foundational data engineering concepts, interview practice, and first-pass design discussions. It is a small model trained on a small corpus, not a comprehensive or authoritative expert. It can omit context, make technical mistakes, and be outdated as product versions change. Verify operational advice against the linked official documentation and your organization's standards. Do not provide credentials, private data, or production secrets in prompts.
The held-out loss is a language-modeling measure on this small Q&A split; it is not a measure of engineering correctness. No broad benchmark or expert review is claimed.
References
Training examples link to official documentation for Apache Airflow, Apache Kafka, Apache Spark, dbt, Apache Iceberg, PostgreSQL, and OpenLineage. The explanations are original summaries, not copied documentation text.