The Rancor is the giant monster kept as a pet by Jabba the Hutt. It lives in a cavernous pit underneath the vile villain's throne room. Anyone who angers or otherwise displeases him is dropped through a convenient trapdoor to be devoured alive by the monster, which is released from its holding cell via an enormous metal door with spikes on the bottom.
We get our first intimation of its existence when Jabba drops Oola the dancing girl down into the pit for reasons that are unclear at best. But even in the Special Edition, which adds a sequence of her down in the pit, we only see the door beginning to open and her death scene is entirely offscreen, and so the Rancor's appearance is held back until much later when Jabba drops heroic Jedi Knight Luke Skywalker down there in an effort to rid himself of the do-gooder. Unfortunately (not that Jabba cares) one of his own guards, Jubnuk, falls in, too.
The Rancor is unleashed and it's one ugly beastie. Try and imagine a pug dog crossbred with a dinosaur and give it really long apelike arms with big clawed hands and you'll be halfway there.
It quickly eats up poor Jubnuk, and, desiring dessert, comes after Luke next. Luke saves himself from sharing Jubnuk's fate by jamming a bone into the Rancor's jaws, forcing its mouth open. Confused and angry, it drops Luke, who takes off running between the Rancor's legs, towards the cell it originally emerged from, as he can see a door used by the keepers to come and go when tending to the beast.
Finally breaking the bone by biting down really, really hard, the Rancor turns and begins to lumber after its really fast food.
Unfortunately for Luke, the door is locked, and the two keepers, Malakili and Giran, aren't too keen on opening the door for him. If anything, they seem surprised to see him and slightly amused. Finding no help there, Luke turns and sees the monster about to come in, pausing to roar dramatically.
Noticing a button to open and close the door nearby, Luke grabs up a skull and hurls it with everything he's got just as the Rancor steps underneath the door. Why the door button is located here and not in the office where the keepers are, and how they're able to come and go from the monster's cell without it eating them, are questions that go unanswered.
Anyway, skull meets button, and the door closes on the Rancor's head, the big spikes on the bottom piercing its head and killing it.
The movie offers us some sympathy for the poor beast, though, in the reaction of its owners. Opening their little door and running in, they're pissed. Giran starts attempting to choke Luke, whilst the other one, Malakili, pushes past them, wide-eyed and stunned, slowly approaching the dead Rancor as though he cannot believe what he's just witnessed.
He begins to cry.
As Luke is taken away to pay for this "outrage," we see Giran putting his arm around his friend and consoling him gently in an alien language. Even monsters have best friends, I guess.