git lfs not working properly for files larger than

问题:

I was suggested by git to use git lfs for large files. After I tracked them with git lfs and checked if they are added to .gitattribute I still get the error that files are larger than 100MB for the same exact files. What are the suggestions here and how I can solve this problem? I would need to upload these large files as part of the project to github as well.

    jalal@klein:~/computer_vision/py-faster-rcnn$ git push -u origin masterUsername for 'https://github.com': monajalal
    Password for 'https://[e-mail]': 
    Counting objects: 65949, done.
    Delta compression using up to 40 threads.
    Compressing objects: 100% (64616/64616), done.
    Writing objects: 100% (65949/65949), 1.53 GiB | 11.18 MiB/s, done.
    Total 65949 (delta 29371), reused 2 (delta 0)
    remote: Resolving deltas: 100% (29371/29371), done.
    remote: error: GH001: Large files detected. You may want to try Git Large File Storage - https://git-lfs.github.com.
    remote: error: Trace: b3969bd24e3d0b92cab7299bbf735509
    remote: error: See http://git.io/iEPt8g for more information.
    remote: error: File caffe-fast-rcnn/boost_1_61_0/bin.v2/libs/math/build/gcc-5.2.1/release/link-static/src/tr1/pch.hpp.gch is 127.21 MB; this exceeds GitHub's file size limit of 100.00 MB
    remote: error: File data/faster_rcnn_models/ZF_faster_rcnn_final.caffemodel is 226.19 MB; this exceeds GitHub's file size limit of 100.00 MB
    remote: error: File data/faster_rcnn_models.tgz is 694.81 MB; this exceeds GitHub's file size limit of 100.00 MB
    remote: error: File data/faster_rcnn_models/VGG16_faster_rcnn_final.caffemodel is 522.92 MB; this exceeds GitHub's file size limit of 100.00 MB
    remote: error: File caffe-fast-rcnn/boost_1_61_0/bin.v2/libs/math/build/gcc-5.2.1/release/src/tr1/pch.hpp.gch is 127.21 MB; this exceeds GitHub's file size limit of 100.00 MB
    To https://github.com/monajalal/faster_RCNN.git
     ! [remote rejected] master -> master (pre-receive hook declined)
    error: failed to push some refs to 'https://github.com/monajalal/faster_RCNN.git'



jalal@klein:~/computer_vision/py-faster-rcnn$ ls
caffe-fast-rcnn  data  experiments  lib  LICENSE  models  README.md  test  tools
jalal@klein:~/computer_vision/py-faster-rcnn$ git lfs track "*.gch"
Tracking *.gch
jalal@klein:~/computer_vision/py-faster-rcnn$ git lfs track "*.tgz"
Tracking *.tgz
jalal@klein:~/computer_vision/py-faster-rcnn$ git lfs track "*.caffemodel"
Tracking *.caffemodel
jalal@klein:~/computer_vision/py-faster-rcnn$ git add -A
jalal@klein:~/computer_vision/py-faster-rcnn$ git commit -m "files larger than 100MB tracked with git lfs"
[master 
b60c05f] files larger than 100MB tracked with git lfs
 6 files changed, 3 insertions(+)
 create mode 100644 .gitattributes
 rewrite caffe-fast-rcnn/boost_1_61_0/bin.v2/libs/math/build/gcc-5.2.1/release/link-static/src/tr1/pch.hpp.gch (99%)
 rewrite caffe-fast-rcnn/boost_1_61_0/bin.v2/libs/math/build/gcc-5.2.1/release/src/tr1/pch.hpp.gch (99%)
 rewrite data/faster_rcnn_models.tgz (99%)
 rewrite data/faster_rcnn_models/VGG16_faster_rcnn_final.caffemodel (99%)
 rewrite data/faster_rcnn_models/ZF_faster_rcnn_final.caffemodel (99%)
jalal@klein:~/computer_vision/py-faster-rcnn$ 
jalal@klein:~/computer_vision/py-faster-rcnn$ ls -a
.  ..  caffe-fast-rcnn  data  experiments  .git  .gitattributes  lib  LICENSE  models  README.md  test  tools


jalal@klein:~/computer_vision/py-faster-rcnn$ git push -u origin master
Username for 'https://github.com': monajalal
Password for 'https://[e-mail]': 
Username for 'https://github.com': monajalal
Password for 'https://[e-mail]': 
Git LFS: (5 of 5 files) 1.66 GB / 1.66 GB                                                                                                             
Counting objects: 65972, done.
Delta compression using up to 40 threads.
Compressing objects: 100% (64633/64633), done.
Writing objects: 100% (65972/65972), 1.53 GiB | 11.18 MiB/s, done.
Total 65972 (delta 29378), reused 2 (delta 0)
remote: Resolving deltas: 100% (29378/29378), done.
remote: error: GH001: Large files detected. You may want to try Git Large File Storage - https://git-lfs.github.com.
remote: error: Trace: 4acfe1a1fdf5874edf9141fa8327428d
remote: error: See http://git.io/iEPt8g for more information.
remote: error: File caffe-fast-rcnn/boost_1_61_0/bin.v2/libs/math/build/gcc-5.2.1/release/link-static/src/tr1/pch.hpp.gch is 127.21 MB; this exceeds GitHub's file size limit of 100.00 MB
remote: error: File data/faster_rcnn_models/ZF_faster_rcnn_final.caffemodel is 226.19 MB; this exceeds GitHub's file size limit of 100.00 MB
remote: error: File data/faster_rcnn_models.tgz is 694.81 MB; this exceeds GitHub's file size limit of 100.00 MB
remote: error: File data/faster_rcnn_models/VGG16_faster_rcnn_final.caffemodel is 522.92 MB; this exceeds GitHub's file size limit of 100.00 MB
remote: error: File caffe-fast-rcnn/boost_1_61_0/bin.v2/libs/math/build/gcc-5.2.1/release/src/tr1/pch.hpp.gch is 127.21 MB; this exceeds GitHub's file size limit of 100.00 MB
To https://github.com/monajalal/faster_RCNN.git
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'https://github.com/monajalal/faster_RCNN.git'
jalal@klein:~/computer_vision/py-faster-rcnn$

Inside .gitattribute file I have:

    *.gch filter=lfs diff=lfs merge=lfs -text
    *.tgz filter=lfs diff=lfs merge=lfs -text
    *.caffemodel filter=lfs diff=lfs merge=lfs -text

回答1:

Your history still contains the large files, which you need to rewrite to remove.

git bfg is a tool (written in java) to easily remove larger files from history.

Using only built-in tools, you can use git filter-branch with --tree-filter to remove the file (examples should be in the doc).

Warning:

Note that tools like BFG and git filter-branch rewrite history. If you rewrite commits that you have pushed, this can affect others who build on those commits. Use with care.