Python

  1. Show all available font families in Python:

     import matplotlib.font_manager
 from IPython.core.display import HTML

 def make_html(fontname):
     return "<p>{font}: <span style='font-family:{font}; font-size: 24px;'>{font}</p>".format(font=fontname)
 code = "\n".join([make_html(font) for font in sorted(set([f.name for f in matplotlib.font_manager.fontManager.ttflist]))])
 HTML("<div style='column-count: 2;'>{}</div>".format(code))

    To see just the names of available font families:

     sorted([f.name for f in matplotlib.font_manager.fontManager.ttflist])
  2. Get immediate subfolders (next function): 
     import os
 folder = '~'
 sfL = next(os.walk(folder))[1]
  3. Bitmap image can be converted to SMOOTH vector image easily using Python. The original image shown below can be smoothed by the following command: 
     import matplotlib.pyplot as plt
 plt.imshow(data, interpolation='spline16')

    Then we need to save the smoothed image (vector) in a vector image format (.pdf, .eps, .svg), using the following command:

     plt.savefig(r'I:\Github\Python\ForFun\Peng\OP\OP_pdf.pdf', format='pdf')

    This is the image where I compare original, pdf and svg savefig outcome: compare

  4. Matplotlib colormap: mpl has a convenient way of creating discrete colormap for curves. more info

     from matplotlib import cm
 viridis = cm.get_cmap('viridis', 8)

    Then the colors in the colormap viridis can be accessed by viridis(x), where x takes value between 0 and 1. For example, viridis(0.5) is (0.122312, 0.633153, 0.530398, 1.0) in RGBA. All colors can bee seen by calling viridis:

    picture 2

    There is no need to import cm again, since it’s already included in matplotlib.pyplot, which I almost import every time. An example:

     import matplotlib.pyplot as plt
 set3 = plt.cm.get_cmap("Set3")

    The “Set3” colormap is created where colors can be accessed by set3().

    set3 cmap

  5. Convert date object to formatted string: (reference)

     Timestamp('2022-01-17 00:00:00') -> '01172022'

    use

     date.strftime("%m%d%Y")

  6. To “stuff” a pandas.DataFrame with np.nan: when calculating MSD from particle trajectories, we want to have frame column to be continuous integer array. However, sometimes in the data, we have frame = [0, 50, 100, ...]. To fill all the frames without position data with np.nan, we can set frame as the index of the DataFrame, then reindex the DataFrame with continuous integers np.arange(...). This creates a trajectory data with continuous frames:

     pos = traj.set_index('frame')[pos_columns]
 pos = pos.reindex(np.arange(pos.index[0], 1 + pos.index[-1]))

    NOTE: I found this method in the trackpy function _msd_gaps(). This has led to a problem when I try to plt.plot the MSD data, because there are many np.nan in the data, and plt.plot cannot show a continuous line. To make the plot out of the MSD data stuffed with np.nan, we need to dropna() first.

  7. A good serif font for plot labels: stix. To apply to all matplotlib text:

     import matplotlib
 matplotlib.rcParams["font.family"] = "STIXGeneral"
 matplotlib.rcParams['mathtext.fontset'] = 'stix'

    Note that both the regular text and the math text need to be configured.

Linux

  1. Disk activity monitor: sudo iotop

Git (GitHub)

  1. To reset .git: sometimes you want to ignore certain files, which are already tracked in the repository. At times like this, you need to reset the .git, which contains the tracking information. First, untrack everything by typing (this removes any changed files from the index(staging area)) 
    git rm -r --cached .

    then run

    git add .

    This adds everything back, but according to the updated .gitignore. Commit this change and it’s done.

  2. Command line authentication: starting from summer 2021, GitHub no longer allow username and password authentication from command line interface. Token authentication becomes required for git operations (original blog). With this requirement implemented, when I try to clone my repositories using git clone on a new, unauthenticated computer, access is denied.

    denied access

    It’s possible to get around this problem with a GUI. In my memory, it was just a set of well-guided steps, involving some clicks in a web browser, and that’s it. However, when a GUI is not available, setting up the authentication is not so straightforward.

    One way to do it when only command line interface is available, is to generate a pair of SSH keys using ssh-keygen. This will generate a pair of key files with encrypted strings.

    example key files

    Paste the content of the pub-key (id_rsa.pub) to the “SSH and GPG keys” tab in GitHub account settings page.

    ssh keys page

    Click on “New SSH key” and paste the pub key in the blank. Now, try to clone the repo again with git clone ssh@url.copied.from.github.

    it works

SSH

  1. ssh works when the device you are connecting to is in the same private network.

  2. VPN tools, such as OpenVPN, can create a virtual private network, allowing connection from outside a private network.

  3. By default, closing a ssh session kills any user programs. To keep programs running after closing a session, we can install a software screen on the Linux system.

  4. Virtual machines can be used to test server behavior.

ffmpeg

  1. Quick start 
    ffmpeg -y -framerate 50 -i %05d.jpg -vcodec h264 output.avi

    -y means automatically replace existing file without asking. -i specifies input images. Note that input images can be an image sequence with C style formatted strings as names. -vcodec provides a lot of options of different encoders, among which h264 is usually a safe choice. Not many encoders does black and white encodings. The last argument is always the output file name.

  2. ffmpeg image sequence formatted input only supports consecutive names, e.g. image0, image1, image2, … Inconsecutive names such as image0, image2, image4 cannot be converted by default settings. On Linux build, there is an optional setting called “Globbing” that can be switched on by passing -pattern_type glob. However, such option is not included on Windows ffmpeg. Source

  3. To convert an image sequence (where the numbering does not start from 0-4) to a video, use the option -start_number before input -i. For example: 
    ffmpeg -framerate 25 -start_number 1001 -i %04d.jpg out.avi
  4. To trim a video, use (time is hh:mm:ss) 
    ffmpeg -i input.mp4 -ss 00:05:20 -t 00:10:00 -c:v copy -c:a copy out.avi

Cloud Storage

  1. Globus file transfer supports command line interface (CLI). Batch transfer with custom filtering can be achieved.