Anaconda is decoding a JSON on each keypress

[hyperknot]

Expected Behaviour

Anaconda should not decode a JSON at every keypress.
It should also not do a directory search up till root on each keypress.

Actual Behaviour

In a project, when an .anaconda file is present, it decodes the file on each keypress to get the settings out.
If the file is not present, it does a directory lookup for such a file, until the / root directory.
Anaconda likes to do this whole thing 5x at start and 2x for each keypress.

Steps to Reproduce

Add sublime.error_message('environfile ' + environfile) after this line:

anaconda/anaconda_lib/helpers.py

Line 213 in a8f6046

environfile = os.path.join(dirname, '.anaconda')

Add sublime.error_message('JSON LOADING .anaconda') after this line:

anaconda/anaconda_lib/helpers.py

Line 218 in a8f6046

data = json.loads(jsonfile.read())

ST3, Anaconda and OS versions

ST: 3143
Anaconda: master
OS: OS X 10.12.6

ST3 Console Logs

https://bpaste.net/show/1be446a4e817

Anaconda's JsonServer Logs

2017-10-27 02:44:31,544: DEBUG   : bind: address=('localhost', 57422)
2017-10-27 02:44:31,544: DEBUG   : listen: backlog=5
2017-10-27 02:44:31,544: INFO    : Anaconda Server started in 57422 for PID 81201 with cache dir /Users/user/Library/Caches/Jedi/project_web and extra paths /Users/user/nonsync/dev/project/project/viewer4/project_web
2017-10-27 02:44:31,621: INFO    : Incomming connection from ('127.0.0.1', 57431)
2017-10-27 02:44:31,622: INFO    : Incomming connection from ('127.0.0.1', 57432)
2017-10-27 02:44:31,629: INFO    : client requests: lint
2017-10-27 02:45:16,673: INFO    : process 81201 does not exists stopping server...
2017-10-27 02:45:17,678: INFO    : Closing the socket, server will be shutdown now...
2017-10-27 02:45:20,255: DEBUG   : bind: address=('localhost', 57437)
2017-10-27 02:45:20,255: DEBUG   : listen: backlog=5
2017-10-27 02:45:20,255: INFO    : Anaconda Server started in 57437 for PID 81458 with cache dir /Users/user/Library/Caches/Jedi/project_web and extra paths /Users/user/nonsync/dev/project/project/viewer4/project_web
2017-10-27 02:45:20,327: INFO    : Incomming connection from ('127.0.0.1', 57443)
2017-10-27 02:45:20,328: INFO    : Incomming connection from ('127.0.0.1', 57444)
2017-10-27 02:45:20,334: INFO    : client requests: lint
2017-10-27 02:46:06,674: INFO    : client requests: doc

[DamnWidget]

Anaconda does't likes to do this thing, is more a matter of anaconda has to do this thing, otherwise changes made to the .anaconda hook file will be silently ignored.

What is your suggestion?

[hyperknot]

I thought the logical thing would be to only read settings once.
I'd just cache all settings per session. If something got changed it'd be normal for the user to reopen the project or restart the editor, wouldn't it?

[hyperknot]

I made a first try at #720, needs more testing. Also, since a subprocess is quite slow, I'm not sure we should add the default values to the cache. I'll also ask Pipenv to add the two information (--py, --venv) in one command, not two, so we don't need two subprocesses.

[jamesandres]

Hello, perhaps this will be useful info to someone. I'm one of the folks who has been plagued by high CPU usage when running the Anaconda plugin in sublime. Generally that issue has been attributed to JEDI.

I've been running the patch in #720 for the last day and my CPU usage is incredibly reduced. Anecdotally st3 CPU usage has gone from ramping up to using 60%/70% of one core to using 15%/20% (while typing and json_server is processing text). Most importantly when I stop typing it then falls back to an idle of 5%.

Worth noting I'm working on several fairly large codebases. ~400K LOC Python, ~90K LOC coffeescript in window 1 and ~700K LOC python in window 2. Yes, I'm excluding node_modules, etc. from the projects.

@DamnWidget perhaps much of the issues with high cpu usage are down to the lack of caching on the config file?

PS: I could come up with some accurate CPU data if there is interest.

[hyperknot]

@jamesandres look the Jedi CPU issue can be fixed by rolling back to 2.1.18.

Remove the contents of the Anaconda folder and do a git clone into there, that way you can easily jump between versions.

[jamesandres]

It's true, I have also found some versions of JEDI are much better than others. But this isn't about JEDI.

I just found it curious that running stock Anaconda with the default version of JEDI has much reduced CPU when some basic caching (ie: #720) is applied to the json unpacking of the config.

[hyperknot]

Yes, I agree, unpacking a JSON on every keypress is absolutely crazy, no matter what Jedi is used. And caching that is such a simple fix actually!

[hyperknot]

@DamnWidget if I remove the pipfile support from #720, would you be willing to merge the JSON caching fix?

[DamnWidget]

@jamesandres do you have an .anaconda config file in your working path?

[jamesandres]

No, well find ~ -name .anaconda turns up nothing anyway

[DamnWidget]

Then how do you explain the CPU improvement? If you don't have a .anconda file in your path, there is no JSON decoding

[hyperknot]

In theory it saves him an if os.path.exists(environfile) and os.path.isfile(environfile): but I doubt it'd give high CPU usage.

@jamesandres can you try 2.1.18?

[DamnWidget]

Let's say that the average user works into an average of five deep levels in the path (and I am being generous here), examples:
/Users/name/Projects/Python/ProjectName
/home/name/development/whatever/project_name
C:\\Users\\name\\Documents\\Programming\\ProjectName

Look for the .anaconda file in a loop of N = 5 average is not the cause of any CPU usage and less taking into account that after the first look, the operating system buffering enters into play.

Can we add lot of extra complexity into this in order to win probably some milliseconds (if any)? Yes
Worth the effort? I am not sure

[hyperknot]

I still don't understand what is the problem of caching it. People almost never change their interpreter path, definitely not between every keypress.

[DamnWidget]

The problem with caching it is that you have to restart ST3 every time that a modification is made to the file, and that is non acceptable. A solution in the middle way (like check is the file last modification time is newer than the time that we read it in order to load it) I could be open to accept.

[jamesandres]

I'll try to prove a bit more concretely what the improvement is. Perhaps it's just luck or some other thing changed.

[hyperknot]

I believe my caching only needs a tab to be reopened, not an ST3 restart.

…

On 24 November 2017 at 12:06, Oscar Campos ***@***.***> wrote: The problem with caching it is that you have to restart ST3 every time that a modification is made to the file, and that is non acceptable. A solution in the middle way (like check is the file last modification time is newer than the time that we read it in order to load it) I could be open to accept. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#719 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAeKjxmguAYJOAYzXUfbG6f0Cbmlr-epks5s5qMhgaJpZM4QIaCF> .

[DamnWidget]

Sorry @hyperknot I didn't read the code deeply, feel free to clean the pipfile parts and write a new comment or mark me as reviewer when done.

[hyperknot]

@DamnWidget OK, I'll make a clean PR and test it properly before submitting it.