What is in your Mathematica tool bag? [closed]

Question

We all know that Mathematica is great, but it also often lacks critical functionality. What kind of external packages / tools / resources do you use with Mathematica?

I'll edit (and invite anyone else to do so too) this main post to include resources which are focused on general applicability in scientific research and which as many people as possible will find useful. Feel free to contribute anything, even small code snippets (as I did below for a timing routine).

Also, undocumented and useful features in Mathematica 7 and beyond you found yourself, or dug up from some paper/site are most welcome.

Please include a short description or comment on why something is great or what utility it provides. If you link to books on Amazon with affiliate links please mention it, e.g., by putting your name after the link.

---

Packages:

1. LevelScheme is a package that greatly expands Mathematica's capability to produce good looking plots. I use it if not for anything else then for the much, much improved control over frame/axes ticks. Its newest version is called SciDraw, and it will be released sometime this year.
2. David Park's Presentation Package (US$50 - no charge for updates)
3. Jeremy Michelson's grassmannOps package provides resources for doing algebra and calculus with Grassmann variables and operators that have non trivial commutation relations.
4. John Brown's GrassmannAlgebra package and book for working with Grassmann and Clifford algebras.
5. RISC (Research Institute for Symbolic Computation) has a variety of packages for Mathematica (and other languages) available for download. In particular, there is Theorema for automated theorem proving, and the multitude of packages for symbolic summation, difference equations, etc. at the Algorithmic Combinatorics group's software page.

Tools:

1. MASH is Daniel Reeves's excellent Perl script essentially providing scripting support for Mathematica v7. (Now built in as of Mathematica 8 with the -script option.)
2. An alternate Mathematica shell with a GNU readline input (using python, *nix only)
3. ColourMaths package allows you to visually select parts of an expression and manipulate them. http://www.dbaileyconsultancy.co.uk/colour_maths/colour_maths.html

Resources:

1. Wolfram's own repository MathSource has a lot of useful if narrow notebooks for various applications. Also check out the other sections such as

   • Current Documentation,
   • Courseware for lectures,
   • and Demos for, well, demos.

2. The Mathematica Wikibook.

Books:

1. Mathematica programming: an advanced introduction by Leonid Shifrin (web, pdf) is a must read if you want to do anything more than For loops in Mathematica. We have the pleasure of having Leonid himself answering questions here.
2. Quantum Methods with Mathematica by James F. Feagin (amazon)
3. The Mathematica Book by Stephen Wolfram (amazon) (web)
4. Schaum's Outline (amazon)
5. Mathematica in Action by Stan Wagon (amazon) - 600 pages of neat examples and goes up to Mathematica version 7. Visualization techniques are especially good, you can see some of them on the author's Demonstrations Page.
6. Mathematica Programming Fundamentals by Richard Gaylord (pdf) - A good concise introduction to most of what you need to know about Mathematica programming.
7. Mathematica Cookbook by Sal Mangano published by O'Reilly 2010 832 pages. - Written in the well known O'Reilly Cookbook style: Problem - Solution. For intermediates.
8. Differential Equations with Mathematica, 3rd Ed. Elsevier 2004 Amsterdam by Martha L. Abell, James P. Braselton - 893 pages For beginners, learn solving DEs and Mathematica at the same time.

Undocumented (or scarcely documented) features:

7. How to customize Mathematica keyboard shortcuts. See this question.
8. How to inspect patterns and functions used by Mathematica's own functions. See this answer
9. How to achieve consistent size for GraphPlots in Mathematica? See this question.
10. How to produce documents and presentations with Mathematica. See this question.

Answer 1

My favorite hacks are small code-generating macros that allow you to replace a bunch of standard boilerplate commands with one short one. Alternatively, you can create commands for opening/creating notebooks.

Here is what I've been using for a while in my day-to-day Mathematica workflow. I found myself performing the following a lot:

1. Make a notebook have a private context, load package(s) I need, make it autosave.
2. After working with this notebook for a while, I'd want to do some throw away scratch computations in a separate notebook, with its own private context, while having access to definitions I've been using in the "main" notebook. Because I set up the private context, this requires to manually adjust $ContextPath

Doing all this by hand over and over is a pain, so let's automate! First, some utility code:

(* Credit goes to Sasha for SelfDestruct[] *)
SetAttributes[SelfDestruct, HoldAllComplete];
SelfDestruct[e_] := (If[$FrontEnd =!= $Failed,
   SelectionMove[EvaluationNotebook[], All, EvaluationCell]; 
   NotebookDelete[]]; e)

writeAndEval[nb_,boxExpr_]:=(
    NotebookWrite[nb,  CellGroupData[{Cell[BoxData[boxExpr],"Input"]}]];
    SelectionMove[nb, Previous, Cell]; 
    SelectionMove[nb, Next, Cell];
    SelectionEvaluate[nb];
)

ExposeContexts::badargs = 
  "Exposed contexts should be given as a list of strings.";
ExposeContexts[list___] := 
 Module[{ctList}, ctList = Flatten@List@list; 
  If[! MemberQ[ctList, Except[_String]],AppendTo[$ContextPath, #] & /@ ctList, 
   Message[ExposeContexts::badargs]];
  $ContextPath = DeleteDuplicates[$ContextPath];
  $ContextPath]

    Autosave[x:(True|False)] := SetOptions[EvaluationNotebook[],NotebookAutoSave->x];

Now, let's create a macro that's going to put the following cells in the notebook:

SetOptions[EvaluationNotebook[], CellContext -> Notebook]
Needs["LVAutils`"]
Autosave[True]

And here's the macro:

MyPrivatize[exposedCtxts : ({__String} | Null) : Null]:=
  SelfDestruct@Module[{contBox,lvaBox,expCtxtBox,assembledStatements,strList},
    contBox = MakeBoxes[SetOptions[EvaluationNotebook[], CellContext -> Notebook]];
    lvaBox = MakeBoxes[Needs["LVAutils`"]];

    assembledStatements = {lvaBox,MakeBoxes[Autosave[True]],"(*********)"};
    assembledStatements = Riffle[assembledStatements,"\[IndentingNewLine]"]//RowBox;
    writeAndEval[InputNotebook[],contBox];
    writeAndEval[InputNotebook[],assembledStatements];
    If[exposedCtxts =!= Null,
       strList = Riffle[("\"" <> # <> "\"") & /@ exposedCtxts, ","];
       expCtxtBox = RowBox[{"ExposeContexts", "[", RowBox[{"{", RowBox[strList], "}"}], "]"}];
       writeAndEval[InputNotebook[],expCtxtBox];
      ]
 ]

Now when I type in MyPrivatize[] is creates the private context and loads my standard package. Now let's create a command that will open a new scratch notebook with its own private context (so that you can hack there with wild abandon without the risk of screwing up the definitions), but has access to your current contexts.

SpawnScratch[] := SelfDestruct@Module[{nb,boxExpr,strList},
    strList = Riffle[("\"" <> # <> "\"") & /@ $ContextPath, ","];
    boxExpr = RowBox[{"MyPrivatize", "[",
        RowBox[{"{", RowBox[strList], "}"}], "]"}];
    nb = CreateDocument[];
    writeAndEval[nb,boxExpr];
]

The cool thing about this is that due to SelfDestruct, when the command runs it leaves no trace in the current notebook -- which is good, because otherwise it would just create clutter.

For extra style points, you can create keyword triggers for these macros using InputAutoReplacements, but I'll leave this as an exercise for the reader.

Answer 2

PutAppend with PageWidth -> Infinity

In Mathematica using of the PutAppend command is the most straightforward way to maintain a running log file with results of intermediate computations. But it uses by default PageWith->78 setting when exporting expressions to a file and so there is no guarantee that every intermediate output will take only one line in the log.

PutAppend does not have any options itself but tracing its evaluations reveals that it is based on the OpenAppend function which has the PageWith option and allows changing its default value by the SetOptions command:

In[2]:= Trace[x>>>"log.txt",TraceInternal->True]
Out[2]= {x>>>log.txt,{OpenAppend[log.txt,CharacterEncoding->PrintableASCII],OutputStream[log.txt,15]},Null}

So we can get PutAppend to append only one line at a time by setting:

SetOptions[OpenAppend, PageWidth -> Infinity]

---

UPDATE

There is a bug introduced in version 10 (fixed in version 11.3): SetOptions no longer affects the behavior of OpenWrite and OpenAppend.

A workaround is to implement your own version of PutAppend with explicit PageWidth -> Infinity option:

Clear[myPutAppend]
myPutAppend[expr_, pathtofile_String] :=
 (Write[#, expr]; Close[#];) &[OpenAppend[pathtofile, PageWidth -> Infinity]]

Note that we also may implement it via WriteString as shown in this answer, but in this case it will be necessary to preliminarily convert the expression into the corresponding InputForm via ToString[expr, InputForm].

Answer 3

Start without a blank notebook open

I was bothered by having Mathematica start with a blank notebook open. I could close this notebook with a script, but it would still flash open briefly. My hack is to create a file Invisible.nb containing:

Notebook[{},Visible->False]

And add this to my Kernel\init.m :

If[Length[Notebooks["Invisible*"]] > 0, 
  NotebookClose[Notebooks["Invisible*"][[1]]]
]

SetOptions[$FrontEnd,
  Options[$FrontEnd, NotebooksMenu] /. 
    HoldPattern["Invisible.nb" -> {__}] :> Sequence[]
]

I now start Mathematica by opening Invisible.nb

There may be a better way, but this has served me well.

---

Customized Fold and FoldList

Fold[f, x] is made equivalent to Fold[f, First@x, Rest@x]

Incidentally, I believe this may find its way into a future version of Mathematica.

Surprise! This has been implemented, though it is presently undocumented. I am informed that it was implemented in 2011 by Oliver Ruebenkoenig, apparently not long after I posted this. Thank you Oliver Ruebenkoenig!

Unprotect[Fold, FoldList]

Fold[f_, h_[a_, b__]] := Fold[f, Unevaluated @ a, h @ b]
FoldList[f_, h_[a_, b__]] := FoldList[f, Unevaluated @ a, h @ b]

(* Faysal's recommendation to modify SyntaxInformation *)
SyntaxInformation[Fold]     = {"ArgumentsPattern" -> {_, _, _.}};
SyntaxInformation[FoldList] = {"ArgumentsPattern" -> {_, _., {__}}};

Protect[Fold, FoldList]

Updated to allow this:

SetAttributes[f, HoldAll]
Fold[f, Hold[1 + 1, 2/2, 3^3]]

f[f[1 + 1, 2/2], 3^3]

---

"Dynamic Partition"

See Mathematica.SE post #7512 for a new version of this function.

Frequently I want to partition a list according to a sequence of lengths.

pseudo-code example:

partition[{1,2,3,4,5,6}, {2,3,1}]

Output: {{1,2}, {3,4,5}, {6}}

I came up with this:

dynP[l_, p_] := 
 MapThread[l[[# ;; #2]] &, {{0} ~Join~ Most@# + 1, #} &@Accumulate@p]

Which I then completed with this, including argument testing:

dynamicPartition[l_List, p : {_Integer?NonNegative ..}] :=
  dynP[l, p] /; Length@l >= Tr@p

dynamicPartition[l_List, p : {_Integer?NonNegative ..}, All] :=
  dynP[l, p] ~Append~ Drop[l, Tr@p] /; Length@l >= Tr@p

dynamicPartition[l_List, p : {_Integer?NonNegative ..}, n__ | {n__}] :=
  dynP[l, p] ~Join~ Partition[l ~Drop~ Tr@p, n] /; Length@l >= Tr@p

The third argument controls what happens to elements beyond the split specification.

---

Szabolcs's Mathematica tricks

The one I use most frequently is the Paste Tabular Data Palette

CreatePalette@
 Column@{Button["TSV", 
    Module[{data, strip}, 
     data = NotebookGet[ClipboardNotebook[]][[1, 1, 1]];
     strip[s_String] := 
      StringReplace[s, RegularExpression["^\\s*(.*?)\\s*$"] -> "$1"];
     strip[e_] := e;
     If[Head[data] === String, 
      NotebookWrite[InputNotebook[], 
       ToBoxes@Map[strip, ImportString[data, "TSV"], {2}]]]]], 
   Button["CSV", 
    Module[{data, strip}, 
     data = NotebookGet[ClipboardNotebook[]][[1, 1, 1]];
     strip[s_String] := 
      StringReplace[s, RegularExpression["^\\s*(.*?)\\s*$"] -> "$1"];
     strip[e_] := e;
     If[Head[data] === String, 
      NotebookWrite[InputNotebook[], 
       ToBoxes@Map[strip, ImportString[data, "CSV"], {2}]]]]], 
   Button["Table", 
    Module[{data}, data = NotebookGet[ClipboardNotebook[]][[1, 1, 1]];
     If[Head[data] === String, 
      NotebookWrite[InputNotebook[], 
       ToBoxes@ImportString[data, "Table"]]]]]}

---

Modify external data from within Compile

Recently Daniel Lichtblau showed this method I had never seen before. In my opinion it significantly extends the utility of Compile

ll = {2., 3., 4.};
c = Compile[{{x}, {y}}, ll[[1]] = x; y];

c[4.5, 5.6]

ll

(* Out[1] = 5.6  *)

(* Out[2] = {4.5, 3., 4.}  *)

Answer 4

General PDF/EMF export problems and solutions

1) It is completely unexpected and undocumented, but Mathematica exports and saves graphics in PDF and EPS formats using a set of style definitions that differs from the one used for displaying Notebooks on screen. By default Notebooks are displayed on screen in the "Working" style environment (which is default value for the ScreenStyleEvironment global $FrontEnd option) but are printed in the "Printout" style environment (which is default value for the PrintingStyleEnvironment global $FrontEnd option). When one exports graphics in raster formats such as GIF and PNG or in EMF format Mathematica generates graphics that looks exactly like it looks inside Notebook. It seems that the "Working" style environment is used for rendering in this case. But it is not the case when you export/save anything in PDF or EPS formats! In this case the "Printout" style environment is used by default that differs very deeply from the "Working" style environment. First of all, the "Printout" style environment sets Magnification to 80%. Secondly, it uses its own values for the font sizes of different styles and this results in inconsistent font size changes in the genarated PDF file as compared with the original on-screen representation. The latter can be called FontSize fluctuations which are very annoying. But happily this can be avoided by setting the PrintingStyleEnvironment global $FrontEnd option to "Working":

SetOptions[$FrontEnd, PrintingStyleEnvironment -> "Working"]

2) The common problem with exporting to EMF format is that most of programs (not only Mathematica) generate a file that looks nice at the default size but becomes ugly when you zoom it in. It is because metafiles are sampled at screen resolution fidelity. The quality of the generated EMF file can be enhanced by Magnifying the original graphical object so that exactness of sampling of the original graphics becomes much more precise. Compare two files:

graphics1 = 
  First@ImportString[
    ExportString[Style["a", FontFamily -> "Times"], "PDF"], "PDF"];
graphics2 = Magnify[graphics1, 10];
Export["C:\\test1.emf", graphics1]
Export["C:\\test2.emf", graphics2]

If you insert these files into Microsoft Word and zoom them in you will see that the first "a" has sawtooth on it while the second has not (tested with Mathematica 6).

Another way through ImageResolution was suggested by Chris Degnen (this option has effect at least starting from Mathematica 8):

Export["C:\\test1.emf", graphics1]
Export["C:\\test2.emf", graphics1, ImageResolution -> 300]

3) In Mathematica we have three ways to convert graphics into metafile: via Export to "EMF" (strongly recommended way: produces metafile with highest possible quality), via Save selection As... menu item (produces much lesser precise figure, not recommended) and via Edit ► Copy As ► Metafile menu item (I strongly recommend against this route).

Answer 5

One of the nice things about the Mathematica notebook interface is that it can evaluate expressions in any language, not just Mathematica. As a simple example, consider creating a new Shell input cell type that passes the contained expression to the operating system shell for evaluation.

First, define a function that delegates evaluation of a textual command to the external shell:

shellEvaluate[cmd_, _] := Import["!"~~cmd, "Text"]

The second argument is needed and ignored for reasons that will become apparent later. Next, we want to create a new style called Shell:

1. Open a new notebook.
2. Select the menu item Format/Edit Stylesheet...
3. In the dialog, beside Enter a style name: type Shell.
4. Select the cell bracket beside the new style.
5. Select the menu item Cell/Show Expression
6. Overwrite the cell expression with the Step 6 Text given below.
7. Once again, select the menu item Cell/Show Expression
8. Close the dialog.

Use the following cell expression as the Step 6 Text:

Cell[StyleData["Shell"],
 CellFrame->{{0, 0}, {0.5, 0.5}},
 CellMargins->{{66, 4}, {0, 8}},
 Evaluatable->True,
 StripStyleOnPaste->True,
 CellEvaluationFunction->shellEvaluate,
 CellFrameLabels->{{None, "Shell"}, {None, None}},
 Hyphenation->False,
 AutoQuoteCharacters->{},
 PasteAutoQuoteCharacters->{},
 LanguageCategory->"Formula",
 ScriptLevel->1,
 MenuSortingValue->1800,
 FontFamily->"Courier"]

Most of this expression was copied directly form the built-in Program style. The key changes are these lines:

 Evaluatable->True,
 CellEvaluationFunction->shellEvaluate,
 CellFrameLabels->{{None, "Shell"}, {None, None}},

Evaluatable enables the SHIFT+ENTER functionality for the cell. Evaluation will call the CellEvaluationFunction passing the cell content and content type as arguments (shellEvaluate ignores the latter argument). CellFrameLabels is just a nicety that let's the user identify that this cell is unusual.

With all of this in place, we can now enter and evaluate a shell expression:

1. In the notebook created in the steps above, create an empty cell and select the cell bracket.
2. Select the menu item Format/Style/Shell.
3. Type a valid operating system shell command into the cell (e.g. 'ls' on Unix or 'dir' on Windows).
4. Press SHIFT+ENTER.

It is best to keep this defined style in a centrally located stylesheet. Furthermore, evaluation functions like shellEvaluate are best defined as stubs using DeclarePackage in init.m. The details of both of these activities are beyond the scope of this response.

With this functionality, one can create notebooks that contain input expressions in any syntax of interest. The evaluation function can be written in pure Mathematica, or delegate any or all parts of the evaluation to an external agency. Be aware that there are other hooks that relate to cell evaluation, like CellEpilog, CellProlog and CellDynamicExpression.

A common pattern involves writing the input expression text to a temporary file, compiling the file in some language, running the program and capturing the output for ultimate display in the output cell. There are plenty of details to address when implementing a full solution of this kind (like capturing error messages properly), but one must appreciate the fact that it is not only possible to do things like this, but practical.

On a personal note, it is features like this that makes the notebook interface the center of my programming universe.

Update

The following helper function is useful for creating such cells:

evaluatableCell[label_String, evaluationFunction_] :=
  ( CellPrint[
      TextCell[
        ""
      , "Program"
      , Evaluatable -> True
      , CellEvaluationFunction -> (evaluationFunction[#]&)
      , CellFrameLabels -> {{None, label}, {None, None}}
      , CellGroupingRules -> "InputGrouping"
      ]
    ]
  ; SelectionMove[EvaluationNotebook[], All, EvaluationCell]
  ; NotebookDelete[]
  ; SelectionMove[EvaluationNotebook[], Next, CellContents]
  )

It is used thus:

shellCell[] := evaluatableCell["shell", Import["!"~~#, "Text"] &]

Now, if shellCell[] is evaluated, the input cell will be deleted and replaced with a new input cell that evaluates its contents as a shell command.

Answer 6

My utility functions (I have these built in to MASH, which is mentioned in the question):

pr = WriteString["stdout", ##]&;            (* More                           *)
prn = pr[##, "\n"]&;                        (*  convenient                    *)
perr = WriteString["stderr", ##]&;          (*   print                        *)
perrn = perr[##, "\n"]&;                    (*    statements.                 *)
re = RegularExpression;                     (* I wish mathematica             *)
eval = ToExpression[cat[##]]&;              (*  weren't so damn               *)
EOF = EndOfFile;                            (*   verbose!                     *)
read[] := InputString[""];                  (* Grab a line from stdin.        *)
doList[f_, test_] :=                        (* Accumulate list of what f[]    *)
  Most@NestWhileList[f[]&, f[], test];      (*  returns while test is true.   *)
readList[] := doList[read, #=!=EOF&];       (* Slurp list'o'lines from stdin. *)
cat = StringJoin@@(ToString/@{##})&;        (* Like sprintf/strout in C/C++.  *)
system = Run@cat@##&;                       (* System call.                   *)
backtick = Import[cat["!", ##], "Text"]&;   (* System call; returns stdout.   *)
slurp = Import[#, "Text"]&;                 (* Fetch contents of file as str. *)
                                            (* ABOVE: mma-scripting related.  *)
keys[f_, i_:1] :=                           (* BELOW: general utilities.      *)
  DownValues[f, Sort->False][[All,1,1,i]];  (* Keys of a hash/dictionary.     *)
SetAttributes[each, HoldAll];               (* each[pattern, list, body]      *)
each[pat_, lst_, bod_] := ReleaseHold[      (*  converts pattern to body for  *)
  Hold[Cases[Evaluate@lst, pat:>bod];]];    (*   each element of list.        *)
some[f_, l_List] := True ===                (* Whether f applied to some      *)
  Scan[If[f[#], Return[True]]&, l];         (*  element of list is True.      *)
every[f_, l_List] := Null ===               (* Similarly, And @@ f/@l         *)
  Scan[If[!f[#], Return[False]]&, l];       (*  (but with lazy evaluation).   *)