Steps

For each task, we iterate the following process.

Run git pull in the terminal to update the local git repository with the remote git repository hosted at GitHub and update the codes in the project folder.
Run dvc commit just in case you have modified files in your local folder. This updates .dvc folder with the information on the modified files.
Run dvc pull in the terminal to update the local dvc repository with the remote dvc repository hosted at AWS s3 and update the data files in the project folder.
1. If the local folder is ahead of the remote repository, dvc pull may remove your latest changes in the data. In that case, dvc asks whether the file can be erased. If you don't think the file should be erased, answer n to stop the process and move to the next step.
2. To maintain the latest changes in your local data, first reflect the local changes to your local dvc repository. If you created a new file in an untracked folder, say, folder/file, run dvc add folder/file. If you created a file in a tracked folder, say, folder/file, run dvc commit. Then, run git commit -am "some message" and run dvc pull.
Do some work.
When your work is done, you commit and push the changes. If you are afraid that the others might have pushed changes to the remote repository, you should first run git pull to update the local git repository. However, do NOT run dvc pull.
1. When you run git pull, the .dvc file is synced with the remote repository. If you dvc pull, the changes in data in your local folder can be overwritten by the remote changes. dvc returns a warning if this happens. If you do not want to overwrite the local changes, make sure to answer "n".
Suppose that some code, say main/clean_data.R is modified by the work.
1. If the file is not yet tracked by git, run git add main/clean_data.R in the terminal to let git track the file.
2. If the file is already tracked by git, go to the next step.
Suppose that some data file, say output/cleaned_data.rds, is generated or modified by the work.
1. Try git pull to update .dvc files.
2. If the file is generated in a folder that is not yet tracked by dvc, run dvc add output in the terminal to let dvc track files in the folder.
3. If the file is in a folder that is already tracked by dvc, run dvc commit in the terminal to reflect the changes in the data to the local repository. If there is a tracked and modified folder, the prompt asks you whether to update the folder. Answer yes if they are the folders you modified.
Run git commit -am "some commit message" in the terminal to commit the file changes to the local git repository. The option -a indicates to commit all changes and -m indicates to add some commit message. You should always add some commit message.
Run git push in the terminal to update the remote git repository hosted at GitHub with the local git repository.
1. If you cannot push because the remote repository is ahead of the local repository, first git pull and then git push. If git pull causes a conflict, you may have to resolve the conflict.
  1. This may be done by manually resolving the conflicting code.
  2. If a file, say fileA.txt, has a conflict and you want to overwrite the local changes with the remote changes, you can run the following command in the terminal:
```
git checkout --theirs fileA.txt
git add fileA.txt
git commit -am "overwrite fileA.txt with theirs"
```
  1. If a file, say fileA.txt, had a conflict and you want to use the local changes, you can run the following command in the termianl:
```
git checkout --ours fileA.txt
git add fileA.txt
git commit -am "used our fileA.txt"
```
Run dvc push in the terminal to update the remote dvc repository hosted at AWS s3 with the local dvc repository.

https://s3-us-west-2.amazonaws.com/secure.notion-static.com/f2242ba7-ec27-44ea-b0eb-92c945fe0c6d/Economics_-_main_-_RStudio_2021-04-05_15-25-30.mp4

Best practices

I strongly recommend to manipulate git from the terminal (git bash in Windows and terminal in macOS), because dvc only runs from the terminal. I strongly recommend not use RStudio git GUI, because it sometimes fails to detect changes in files.
The timing of commit should be minimal and complete.
- Minimal means to commit after the completion of each minimal task. By doing so, it gets easier to track histories.
- Complete means to commit only after the goal of the task is achieved.
The commit message should describe the reason for changes rather than the changes themselves because changes can be easily reviewed on GitHub.

Steps

Best practices

Troubleshooting

YAML file structure is corrupted