My Machine Learning Development Environment

29 November 2022

This is a living piece about how I am processing data and training machine learning models with that data specifically for My MastersMy Masters
I am currently doing my masters by research in electrical and information engineering at the University of the Witwatersrand. I am fortunate to be supervised by Professor Ken Nixon and Dr Martin Be...
.

Github

Package & Environment Management

Mention conda env I ran into multiple issues trying to use Build essentials

conda env export > environment.yml

conda-minify https://github.com/jamespreed/conda-minify/issues/11#issuecomment-1023317232

Helps with simplifying environment

Installing tensor-flow with conda caused issues

conda install -c conda-forge mamba
mamba install -c conda-forge tensorflow-gpu 

Mamba was faster but then got error about cuda not being provided

Checking for cuda??

nvcc --version

Now trying this

conda env config vars set LD_LIBRARY_PATH=$CONDA_PREFIX/lib

Cluster

Github Python vs Python notebooks Git setup to clear output Conda and conda environment files https://tdhopper.com/blog/my-python-environment-workflow-with-conda

conda config –remove channels conda-forge conda config –add channels conda-forge