[Scons-users] scons-4.3.0 bug reading scons-3.1.2 generated .sconsign.dblite
Jay West
jhdub23 at gmail.com
Thu Mar 24 03:22:32 EDT 2022
"Deploy" was a poor choice of words. We are a large organization with a
pipeline of builds and quality. Our fastest turnaround/lowest quality
guarantees are quick development builds for preliminary testing of targeted
features (non-production use, obviously). We are long-time users of scons
and rely on the build correctness that scons can usually provide. I
haven't seen another build tool that tries as hard as scons to encode even
the commands and the build environment as dependencies. That's why I was
surprised that a DB compatibility check wasn't in the code.
I think an alternative solution of writing the scons version, or some other
compatibility version information, as a header in the .sconsign.dblite
would be much better than renaming files. If the header is not recognized
as being compatible, nuke the file and start over. Is a reproducer really
needed? Even conceptually, an incompatible DB format between versions is a
known weakness.
Thanks,
Jay
On Tue, Mar 22, 2022 at 10:44 PM Bill Deegan <bill at baddogconsulting.com>
wrote:
> Jay,
>
> You're deploying non-full builds (incremental) to the field? (built in the
> same sandbox as older versions?)
> I would not consider that a good practice.
>
> If we change the default sconsign naming we need to go through a
> deprecation cycle (we need to notify on next release, and then roll a major
> release to push the change), it will break lots of tests, and also affect
> users expecting the sconsign file naming. And even if we did this if you
> used a python 3.5+ with scons 3.0+ and did the same switch back and forth
> that you're doing, the issue would not be resolved.
>
> That said you are free to enable switching back and forth in your code by
> naming the sconsign file to include the python version.
> This would solve your issue.
>
> Could you make a small reproducer? Perhaps there's another way to avoid
> the bad outcome.
>
> -Bill
>
>
> On Mon, Mar 21, 2022 at 11:56 AM Jay West <jhdub23 at gmail.com> wrote:
>
>> I'm not asking for the sconsign file to be made compatible between
>> versions. What I'm asking for is to ensure that scons refuses to create
>> bad builds. A full rebuild is acceptable. Error'ing out is acceptable.
>> Silently creating a bad build is not acceptable.
>>
>> A developer may check out old code that must be run with scons2
>> (scons-3.1.2 with python2). The developer may also check out the most
>> recent code that must run with scons3 (scons-4.3.0 with python3). The
>> following happens:
>>
>>
>> - Compile new code with scons3, followed by compiling old code with
>> scons2. scons2 errors out with a pickling error. This is perfectly fine.
>> The developer removes the .sconsign.dblite file and all is good.
>> - Compile old code with scons2, followed by compiling new code with
>> scon3. The build is successful, but the build is corrupt. *This is
>> not acceptable.*
>>
>>
>> We have unknowingly deployed bad builds to the field, and this is causing
>> all sorts of havoc. I have added the below to our SConstruct file for one
>> project. Can something like this be added to scons?
>>
>> Thanks,
>>
>> Jay
>>
>> def fix_scons_db():
>> '''
>> Fix scons2 vs scons3 DB backward compatibility problem
>>
>> The dependency database from scons-3.1.2 cannot be read by
>> scons-4.3.0. Instead of generating an error, scons-4.3.0 will blissfully
>> ignore the error and produce an incorrect build.
>> In order to workaround this, we'll use use an scons
>> version-specific DB file. Credit for the idea to:
>>
>> https://github.com/Autodesk/arnold-usd/pull/116/commits/a404d5f3400437beb4bad561ef1811538f7af022
>> This solution also future proofs against further scons upgrades.
>> '''
>> scons_db_file_name = '.sconsign.{}'.format(SCons.__version__)
>> SConsignFile(scons_db_file_name)
>> scons_db_file_name += '.dblite' # scons adds this suffix
>> scons_old_db_files = glob.glob('.sconsign.*')
>> if scons_db_file_name in scons_old_db_files:
>> scons_old_db_files.remove(scons_db_file_name)
>> #print('scons DB file name:', scons_db_file_name, 'removing',
>> scons_old_db_files)
>> if len(scons_old_db_files):
>> subprocess.run(['rm -f .sconsign.*'], check=True, shell=True)
>>
>>
>> On Sun, Mar 20, 2022 at 9:36 PM Bill Deegan <bill at baddogconsulting.com>
>> wrote:
>>
>>> There's no (reasonable) way to make the sconsign files be compatible
>>> between py3.5 and p2.7.
>>> The only way to not have invalid (as in crashes scons per your original
>>> email) is to name the sconsign files per python version.
>>> (Which I mentioned in my previous email, and is what the commit you
>>> pointed to).
>>>
>>> Of course as you discovered (most likely) you will rebuild items which
>>> were not out of date.
>>>
>>> The recommended way is to just validate the new scons & python
>>> combination on your build and then make the switch.
>>>
>>> -Bill
>>>
>>>
>>> On Sun, Mar 20, 2022 at 7:23 PM Jay West <jhdub23 at gmail.com> wrote:
>>>
>>>> Actually, the workaround isn't sufficient. We also need to delete all
>>>> the other versions of .sconsign.dblite* as they will be out of date.
>>>>
>>>> Jay
>>>>
>>>>
>>>> On Sun, Mar 20, 2022 at 6:58 PM Jay West <jhdub23 at gmail.com> wrote:
>>>>
>>>>> If scons did a full rebuild, it would not be a problem. The issue is
>>>>> that scons does NOT do a full rebuild, thinking that targets were up to
>>>>> date when they were really not.
>>>>>
>>>>> I see a workaround that someone else took:
>>>>> https://github.com/Autodesk/arnold-usd/pull/116/commits/a404d5f3400437beb4bad561ef1811538f7af022.
>>>>> I will be putting this into our SContruct file. Can this be made the
>>>>> default behavior?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Jay
>>>>>
>>>>>
>>>>> On Sat, Mar 19, 2022 at 11:09 AM Bill Deegan <
>>>>> bill at baddogconsulting.com> wrote:
>>>>>
>>>>>> Switching from py2.7 -> py 3.5 invalidates the .sconsign files
>>>>>> contents.
>>>>>> You have to do a full build when you make that switch.
>>>>>> It's in the release notes somewhere.
>>>>>>
>>>>>> https://scons.org/scons-300-is-available.html
>>>>>>
>>>>>> There's no way around it.
>>>>>> The best you can do is name your SConsign file to include the python
>>>>>> version so at least they won't be invalid when you switch from one to
>>>>>> another, but you will see additional rebuilds which may not be strictly
>>>>>> necessary.
>>>>>>
>>>>>> -Bill
>>>>>>
>>>>>> On Fri, Mar 18, 2022 at 5:03 PM Jay West <jhdub23 at gmail.com> wrote:
>>>>>>
>>>>>>> Yes, our scons-3.1.2 was using python-2.7.17, and our scons-4.3.1 is
>>>>>>> using python-3.9.10. We get a pickling error when trying to run
>>>>>>> scons-3.1.2 after scons-4.3.1 which is good. Running scons-4.3.1 after
>>>>>>> scons-3.1.2 runs without error, but results in bad builds, which is
>>>>>>> terrible.
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Mar 18, 2022 at 3:04 PM Bill Deegan <
>>>>>>> bill at baddogconsulting.com> wrote:
>>>>>>>
>>>>>>>> Sounds like you also switched from python 2.7 to python 3.5 or
>>>>>>>> higher?
>>>>>>>>
>>>>>>>> On Fri, Mar 18, 2022 at 2:07 PM Jay West <jhdub23 at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> We are migrating from scons-3.1.2 to scons-4.3.0 and have
>>>>>>>>> encountered incorrect builds.
>>>>>>>>>
>>>>>>>>> After building with scons-3.1.2, changing some files, and then
>>>>>>>>> running scons-4.3.0, it appears that scons-4.3.0 cannot read some
>>>>>>>>> dependencies from the DB file. Instead of error'ing out or ignoring the DB
>>>>>>>>> file, it happily generates an incorrect build. However, running:
>>>>>>>>>
>>>>>>>>> sconsign .sconsign.dblite
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> gives
>>>>>>>>>
>>>>>>>>> TypeError: '<' not supported between instances of 'str' and 'bytes'
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Has anyone seen this issue?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Jay
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Scons-users mailing list
>>>>>>>>> Scons-users at scons.org
>>>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Scons-users mailing list
>>>>>>>> Scons-users at scons.org
>>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Scons-users mailing list
>>>>>>> Scons-users at scons.org
>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>
>>>>>> _______________________________________________
>>>>>> Scons-users mailing list
>>>>>> Scons-users at scons.org
>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>
>>>>> _______________________________________________
>>>> Scons-users mailing list
>>>> Scons-users at scons.org
>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>> _______________________________________________
>> Scons-users mailing list
>> Scons-users at scons.org
>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>
> _______________________________________________
> Scons-users mailing list
> Scons-users at scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist4.pair.net/pipermail/scons-users/attachments/20220324/54ca71a0/attachment.htm>
More information about the Scons-users
mailing list