CSV Dedupe Script With a Dry-Run Flag

scripts/dedupe_csv.py holds a half-written skeleton of a deduplication utility for customer contact CSV exports.

You have already imported the csv and argparse standard libraries, and left comments marking where you need to fill in the remaining functionality to meet your team’s requirements. The tool needs to process CSV exports pulled from the company’s marketing automation platform, remove duplicate entries, and avoid accidental data loss by letting you preview changes before writing to disk.

Define core deduplication logic

You build the duplicate detection function first, since it forms the backbone of the entire tool. You set the default match keys to email and phone number, per the team’s data governance rules, and add an optional parameter to let users pass custom match keys if they are processing a CSV with different field names. You store seen combinations of match key values in a set, since set lookups are O(1) and keep the script fast even for very large CSV files. You add a counter to track how many duplicates are found, and a list to store the full duplicate rows for logging purposes. Illustrative example: A 12,742-row CSV of Q3 lead submissions typically holds 890 to 1,200 duplicate entries from users re-submitting forms multiple times to access gated content, so the set-based lookup keeps processing time under 2 seconds even for files of that size. You test the function inline with a small sample array of row data to confirm it flags exact matches on the specified keys and ignores rows with only partial matches, like the same email but a different phone number for a user who updated their contact information.

Add dry-run flag parsing and validation

Next you update the argparse configuration to add the –dry-run flag, a boolean argument that defaults to False if not passed. You add validation to check that the input CSV path exists before running any processing, and that the output path is not the same as the input path to avoid overwriting the original file accidentally. You add a check that prints a warning if the output path already exists, and a prompt to confirm overwrite unless the user passes the optional –force flag you add alongside the dry-run option. You add a line that prints a clear header at the start of every run stating whether it is a dry run or a live run, so there is no ambiguity about what the script is doing for anyone running it. This script aligns with the lightweight utility guidelines shared in Build Next Stack’s Python best practices reference.

Test edge cases with sample input

You pull a sample CSV from the marketing team’s shared drive to test the script with real-world data, not just your test dummy set. First you run it with the –dry-run flag enabled to confirm it correctly counts duplicates. You see that it flags 412 duplicates in the 5,200 row sample file, which matches the manual count the marketing intern ran last week to validate their own manual deduplication work. You test the custom match key parameter next, running the script with –match-keys company_name address to deduplicate a CSV of vendor contacts, and confirm it correctly flags 17 duplicate entries for vendors that are listed multiple times with different point of contact names. You test the overwrite warning by pointing the output path to an existing file, and confirm the prompt appears and blocks write operations unless you pass –force or type Y to confirm. You also test the script with an empty CSV, a CSV with only headers, and a CSV where every row is a duplicate, to make sure it handles all edge cases without throwing uncaught errors. You add try-except blocks for common errors like missing columns in the input CSV, and return clear, actionable error messages that tell the user exactly what is wrong, like “Input CSV is missing required match key: phone” instead of a generic KeyError traceback.

Write formatted output for both run modes

You update the logging logic to format output clearly for both run modes. For dry runs, you print a table of the first 20 duplicate rows to the console, so the user can quickly review if the match logic is working as expected, and add a note if there are more than 20 duplicates to view the full list in the temporary log file the dry run writes to the system temp directory. For live runs, you write the cleaned CSV to the specified output path, and save a separate log file with the same name as the output file plus a .duplicates.log suffix that contains all the duplicate rows, the timestamp of the run, and the match keys used, so the team can audit deduplication work later if needed. You add a final summary line that prints at the end of every run, stating the number of rows processed, the number of duplicates found, the number of unique rows written (for live runs) or that would be written (for dry runs), and the file paths for all output files. You make sure all output is human-readable, not just machine-readable, so non-technical team members can run the script and understand the results without needing to ask engineering for help.

Open this file tomorrow to add batch processing support

You save your current progress and leave a comment at the top of the file listing the remaining feature request the operations team submitted: support for batch processing entire directories of CSV files instead of just single files. When you open this file tomorrow, you will first add a new –directory flag that accepts a folder path instead of a single input file path. You will add logic to iterate over every .csv file in the specified directory, run the deduplication logic on each one, and save all cleaned files to a separate output directory with the same folder structure as the input. You will add a configuration option to let users set default match keys, dry run mode, and archive preferences in a config.yaml file so they don’t have to pass the same flags every time they run the script for routine batch processing. You will also add support for exporting the duplicate log as a separate CSV instead of just a plain text file, so the operations team can import it directly into their spreadsheet tool for review without reformatting. You leave a TODO comment in the argparse section marking where you will add the new directory flag, and a note with the path to the sample batch directory the operations team shared for testing. You confirm the current version of the script works as expected for single file processing, both in dry run and live mode, and document the usage instructions in a comment at the top of the file so anyone else on the team can run it without needing to ask you for guidance. You run one final test with the sample marketing CSV, first in dry run mode to confirm the preview works, then in live mode to confirm the output file is correctly formatted and free of duplicates, before closing the file for the day. (Word count: 1372)