Brackets, which come in many types, are used to group things. In mathematics, they may be called delimiters, enclosures or fences. They are never italicised. They are nested in the order {[()]}.
\[ x + \left\{ 1+ \left[ 3+ 2 \times \left( 4 + 5 \right) \right] \right\} \]
Take care with fences when writing mathematics inline.
The equation
\[ z=\frac{x-2}{y} \]
is the same as \(z=\left(x-2\right)/y\). It is not the same as \(z=x-2/y\). In the second case, \(x\) is not being divided by \(y\).
Do not use spaces between numbers or variables and brackets:
\( 2(a-b) \)
not
\( 2~(a-b) \)
Except for a very few specialist cases, an opened fence must be closed by another of the same type, and they must be nested correctly.
Use full-height fences for displayed equations. This requires using an equation editor:
\( \left(\frac{A+B}{A} \right) \) not \( ( \frac{A+B}{A} ) \)
\( \left[ \frac{A+B}{A}+\ln\left( C+D\right) \right] \) not \( [ \frac{A+B}{A}+\ln ( C+D ) ] \)