Making a new project¶
Once you have a notebook that you think it is ready to be its own project you can follow these steps to get it set up. For the examples I’ll use an example project named “bears”:
1. Move the notebook into a new directory with same name¶
mkdir bears mv bears.ipynb ./bears cd bears
NOTE: If your project includes a file called index.ipynb, on the website, that will be treated as a special landing page for the whole project. This can be useful for projects with many notebooks that are best read in a particular order.
2. Start specifying the package dependencies¶
It can take a while to be sure you’ve captured every dependency, but I
usually start by using nbrr (
conda install -c conda-forge nbrr)
which reads the notebooks and looks for dependencies:
nbrr env --directory "." --name bears > anaconda-project.yml
NOTE: We tend to add
nomkl to the list of dependencies to speed
up environment build times. But there is no rule that you must do this.
MKL is used for better runtime performance in numpy operations, but since we
use Numba for most of the internal computations it’s not as important
for these particular projects.
3. Create the anaconda-project file¶
Copy template/.projectignore and template/anaconda-project.yml to your own project, then just replace NAME, DESC, MAINTAINERS, CREATED, LABELS and add the dependencies from step 2.
Labels Labels are used to give others a quick sense of what packages a particular project showcases. They are also used to indicate if a project depends on channels other than defaults. For example:
labels: - channel_conda-forge - datashader - panel
Once the dependencies are sorted, make sure to add the appropriate commands.
If your project contains notebooks, then add a
commands: notebooks: notebook: .
If your project also contains servable panel dashboards then suffix them with “_panel” and add the following to the command dictionary:
dashboard: unix: panel serve *_panel.ipynb supports_http_options: true
NOTE: If your project contains an index.ipynb which illustrates the best path through the notebooks, then instead of the notebooks command above, add the following:
notebook: notebook: index.ipynb
Depending on unreleased package versions
In some cases you may have a notebook that relies on a development
version of a package, or perhaps you wish to refer to a particular git
tag that has not made it into a released version. In this case, you can
pip subsection to your list of dependencies of the form:
- pip: - git+https://github.com/ORG/REPO.git@REF#egg=PACKAGE
ORG is the GitHub organization (or username),
REPO is the name of the
REF is a git reference (e.g a git tag or simply
master to point to the very latest version) and
PACKAGE is the
name of the corresponding Python package. This syntax will use pip to
fetch the necessary code, check out the specified git reference, and
install the package.
Special website building options
If you’d like certain notebooks to be rendered on the website, but not linked
from the main page (perhaps they are linked from other notebooks), then add
the filenames to a list of
orphans: - appendix.ipynb
If you’d like notebooks to be skipped entirely when building the website, use the
skip: - data_prep.ipynb
4. Make sure it works¶
anaconda-project run test
You might need to declare extra dependencies or add data downloads (see bay_trimesh for an example of downloading data).
5. For remote or large data (optional)¶
Unless your data is small enough that it can be processed on every
continuous-integration build, you should make a much smaller version
of the data and put it in
test_data/bears. This step allows automated tests to be run in a
practical way, exercising all of the example’s functionality but on a
feasible subset of the data involved.
6. If using intake (optional)¶
The intake catalog should be at the top level of the project directory and called “catalog.yml”.
bears ├── anaconda-project.yml ├── bears.ipynb └── catalog.yml
If using the intake cache, point the cache to the data dir in the project by defining the INTAKE_CACHE_DIR variable in the anaconda-project file:
variables: INTAKE_CACHE_DIR: data
This way when the user runs the notebook, they will still be able to see the data from within the project directory:
bears ├── anaconda-project.yml ├── bears.ipynb ├── catalog.yml └── data └── f890ce4d538240e87ede9d31a6541443 └── data.csv
Make sure to make a test catalog and put it in
7. Add the project to travis¶
Once everything is set up, add your project to
the pattern that the other projects use. There are two places where you will
have to put it. One for testing:
- <<: *test_project env: DIR=bear
And one for building the project for the website:
- <<: *build_project env: DIR=bear
NOTE: If your project takes a very long time (~15min) to run or requires very
large data (~3GB), you might want to build the project locally on your machine
and check in the result rather than building on CI. In this case replace
build_project above with
local_project and follow the steps under “Building
a project locally”
8. Add thumbnails (optional)¶
By default, when the website is built on travis, a thumbnail is generated for each project. The thumbnail is taken from the first image that the notebook produces. If you’d rather use a different image for a particular notebook: name the image to match the name of the notebook and include it in a “thumbnails” directory within your project. This image must be a png and have the extension “.png”.
Uploading to AE¶
You can upload and deploy any project in Anaconda Enterprise, which is the server we use to host our public Python-backed examples:
cd bears anaconda-project archive bears.zip
Then in the AE interface select “Create”, then “Upload Project” and navigate to the zip file. Once your project has been created, you can deploy it.
NOTE: Dashboard commands should be deployed at <project>.pyviz.demo.anaconda.com and notebooks command at <project>-notebooks.pyviz.demo.anaconda.com
Building a project for the website¶
Most of the projects are built for the website when a special commit
message is passed to Travis CI. The commit message should include the
word “build” and the name of the desired project for example:
commit -m "Fixing typo [build:bears]". If step 7 was done properly,
then this should trigger a Travis CI job that downloads the real data,
sets up the environment, archives the project, then uses nbsite to generate
a thumbnail and evaluated versions of all the notebooks in the project.
Those assets are then stored on the
evaluated branch of the github repo.
After that job completes, another job will start that builds html versions
of all the saved notebooks deploys them to the
gh-pages branch. After that
job has completed, the new content will be visible on the site.
Building a project locally¶
In a minority of cases, the project takes so long to build or the data are
so large, that it isn’t feasible to build the website version of the project
on Travis CI. In those cases, the project maintainer is responsible for
running the build commands locally and committing the results to the
evaluated branch. To build the project follow these steps:
export DIR=bears doit archive_project --name $DIR anaconda-project prepare --directory $DIR conda activate $DIR/envs/default && pip install pyctdev conda install -y -c pyviz/label/dev nbsite sphinx_pyviz_theme selenium phantomjs lxml doit build_project --name $DIR
You should end up with a new directory in the doc dir with the same name as your project. The structure of that directory should be as follows:
doc/bears ├── bears.ipynb ├── bears.rst ├── bears.zip └── thumbnails └── bears.png
Commit only that doc/bears directory to the
evaluated branch. The easiest way to
do that is by moving it to a temporary directory, checking out the
branch and then moving it back:
mv ./doc/$DIR ./tmp git checkout evaluated git pull if [ -e ./doc/$DIR ]; then rm -rf ./doc/$DIR; fi mkdir ./doc/$DIR mv ./tmp/* ./doc/$DIR git add ./doc/$DIR git commit -m "adding $DIR" git push