category object in an (∞,1)-category, groupoid object
The notion of a category can be formulated internal to any other category with enough pullbacks. By regarding groups as one-object (delooping) groupoids, this generalizes the familiar way in which, for instance
topological groups are groups internal to topological spaces
Lie groups are groups internal to smooth manifolds.
An ordinary small category is a category internal to Set.
There is a more general notion of an internal category in a monoidal category, where the pullbacks are replaced by cotensor products.
Let $A$ be any category. A category internal to $A$ consists of
together with
an identity-assigning morphism $e: C_0 \to C_1$;
a composition morphism $c: C_1 \times_{C_0} C_1 \to C_1$;
such that the following diagrams commute, expressing the usual category laws:
Here, the pullback $C_1 \times_{C_0} C_1$ is defined via the square
Notice that inherent to this definition is the assumption that the pullbacks involved actually exist. This holds automatically when the ambient category $A$ has finite limits, but there are some important examples such as $A =\,$ Diff where this is not the case. Here it is helpful to assume simply that $s$ and $t$ have all pullbacks; in the case of $Diff$ this occurs if they are submersions.
A groupoid internal to $A$ is all of the above
such that the cartesian product $C_1 \times C_1$ exists
with a morphism
such that
and
Functors between internal categories are defined in a similar fashion. See functor. But if the ambient category does not satisfy the axiom of choice it is often better to use anafunctors instead; this makes sense when $C$ is a superextensive site.
If $A$ has all pullbacks, then we can form the bicategory $Span(A)$ of spans in $A$. A category in $A$ is precisely a monad in $Span(A)$. The underlying 1-cell is given by the span $(s,t) : C_0 \leftarrow C_1 \to C_0$, and the pullback $C_1 \times_{C_0} C_1$ is the vertex of the composite span $(s,t) \circ (s,t)$. The morphisms $e$ and $c$ are required to be morphisms of spans, which is equivalent to imposing the source and target axioms above. Finally the unit and associativity axioms for monads imply those above.
This approach makes it easy to define the notion of internal profunctor.
The notion of nerve of a small category can be generalised to give an internal nerve construction. For a small category, $D$, its nerve, $N(D)$, is a simplicial set whose set of $n$-simplices is the set of sequences of composable morphisms of length $n$ in $D$. This set can be given by a (multiple) pullback of copies of $D_1$. That description will carry across to give a nerve construction for an internal category.
If $C$ is an internal category in some category $A$, (which thus has, at least, the pullbacks required for the constructions to make sense),its nerve $N(C)$ (or if more precision is needed $N_{int}(C)$, or similar) is the simplicial object in $A$ with
and so on. Face and degeneracy morphisms are induced from the structural moprhisms of $C$ in a fairly obvious way.
Internal functors between internal categories induce simplicial morphisms between the corresponding nerves.
One can also look at this in higher category theory and consider internal n-categories. See
A small category is a category internal to Set. In this case, $C_0$ is a set of objects and $C_1$ is a set of morphisms and the pullback is a subset of the Cartesian product.
Historically, the motivating example was (apparently) the notion of Lie groupoids: a small Lie groupoid is a groupoid internal to the category Diff of smooth manifolds. This generalises immediately to a smooth category?. Similarly, a topological groupoid is a groupoid internal to Top. (Warning: the term ‘topological category’ usually means a topological concrete category, an unrelated notion. Sometimes a ‘topological category’ is defined to be a $Top$-enriched category, which is a special case of the internal definition if it is interpreted strictly and the collection of objects is small.) In these examples, $C_0$ is a “space of objects” and $C_1$ a “space of morphisms”.
Further examples:
A category internal to Set is a small category
A groupoid internal to definable sets is a definable groupoid.
A pointed one-object category internal to Ho(Top) is an H-monoid.
A pointed one-object groupoid internal to Ho(Top) is an H-group.
A cocategory in $C$ is a category internal to $C^{op}$.
A double category is a category internal to Cat.
A double bicategory is a category internal to Bicat (in a suitably weak sense).
A crossed module is equivalent to a category internal to Grp.
A Baez-Crans 2-vector space is a category internal to Vect.
A 2-group is an internal category in Grp and so has an internal nerve, which is a simplicial object in Grp, that is a simplicial group. If the 2-group corresponds to a crossed module, $(C\stackrel{\delta}{\to}P)$, then the simplicial group nerve of $(C\stackrel{\delta}{\to}P)$ has Moore complex having $P$ in dimension 0, and $C$ in dimension 1, with the trivial group in all other dimensions. The only possible non-trivial boundary map from dimension 1 to dimension 0 is then the boundary $\delta$ of the crossed module.
If the ambient (finitely complete) category $\mathbf{E}$ is a cartesian closed category, then the category $Cat(\mathbf{E})$ of categories internal to $\mathbf{E}$ is also cartesian closed. This was proved twice by Charles and Andrée (under her maiden name Bastiani) Ehresmann using generalised sketches, or may be proven directly as follows (see also Johnstone, remark after B2.3.15):
Let $\mathbf{E}$ be a finitely complete cartesian closed category. Then the category $Cat(\mathbf{E})$ of internal categories in $E$ is also finitely complete and cartesian closed.
First suppose $\mathbf{E}$ is finitely complete. Then the category of directed graphs $\mathbf{E}^{\bullet \stackrel{\to}{\to} \bullet}$ is also finitely complete, and since $Cat(\mathbf{E})$ is monadic over $\mathbf{E}^{\bullet \stackrel{\to}{\to} \bullet}$, it follows that $Cat(\mathbf{E})$ is also finitely complete.
Now suppose that $\mathbf{E}$ is finitely complete and cartesian closed. Let $\Delta_3$ denote the category of nonempty ordinals up to and including the ordinal with 4 elements. We have a full and faithful embedding
where the codomain category is cartesian closed. Indeed, the exponential of two objects $F$, $G$ in $\mathbf{E}^{\Delta_3^{op}}$ may be computed as an $\mathbf{E}$-enriched end
when evaluated at $m \in Ob(\mathbf{E}^{\Delta_3^{op}})$, as is easily checked (see for instance here); note that this end is a finite limit diagram since $\Delta_3$ is finite.
If $C$, $D$ are internal categories in $\mathbf{E}$, seen as functors $\Delta_3^{op} \to \mathbf{E}$, the exponential $N C^{N D}$ defines the exponential in $Cat(\mathbf{E})$. To see this, it suffices to check that $N C^{N D}$, as defined by the end formula above, is a category $B$, i.e., is in the essential image of the nerve functor. For in that case, we have natural isomorphisms
whence $B$ satisfies the universal property required of an exponential.
Objects in the essential image of the nerve $N$ are characterized as functors $\Delta_3^{op} \to \mathbf{E}$ which take intervalic joins in $\Delta_3$ to pullbacks in $\mathbf{E}$, as given precisely by the Segal conditions. The remainder of the proof is then finished by the following lemma.
If $C \colon \Delta_3^{op} \to \mathbf{E}$ satisfies the Segal conditions and $X \colon \Delta_3^{op} \to \mathbf{E}$ is any functor, then $C^X$ also satisfies the Segal conditions.
For any $X$ we have the formula
Since the enriched end and the internal hom-functor $Hom(X k, -)$ both preserve pullbacks, we are reduced to checking that
as a functor $\Delta_3^{op} \to \mathbf{E}$ in the argument $m$ (for each fixed $k$).
Note that the displayed statement is a proposition in the language of finitely complete categories (i.e., in finitary essentially algebraic logic). Since hom-functors $\mathbf{E}(e, -) \colon \mathbf{E} \to Set$ jointly preserve and reflect the validity of such propositions, it suffices to prove it for the case where $\mathbf{E} = Set$. But this is classical elementary category theory; it says precisely that if $C$ is a small (ordinary) category, then the usual functor categories $C^{\mathbf{2}}$, $C^{\mathbf{3}}$ are equivalently described by exponentials of (truncated) simplicial sets. This completes the proof.
If the ambient category is a topos, then with the right kind of notion of internal functor, the internal groupoids form the corresponding (2,1)-topos of groupoid-valued stacks and the internal categories form the corresponding 2-topos of category-valued stacks/2-sheaves.
For the precise statement see at 2-topos – In terms of internal categories
A survey with an eye towards Lie groupoids is in
Discussion in terms of monads in spans is in
A detailed discussion with emphasis on the correct anafunctor morphisms between internal categories is in
Discussion with emphasis on topos theory is in section B.2.3 of
and in section V.7 of
An introduction is also in
The original proofs that the category of internal categories is cartesian closed when the ambient category is finitely complete and cartesian closed are in
and in
An old discussion on variants of internal categories, crossed modules and 2-groups is archived here.