Prepare barbecue (medium heat)—place potatoes in a 13x9x2-inch baking dish. Add 1/4 cup oil, chopped fresh rosemary, brown sugar, garlic powder, and cayenne pepper to the potatoes. Sprinkle potatoes with salt and pepper; toss to coat. Brush the grill lightly with oil. Place potatoes on the grill, spacing about 1 inch apart. Grill until potatoes are tender and slightly charred, occasionally turning, about 10 minutes total. Transfer potatoes to bowl, season to taste with salt and pepper, and serve.